Non-convex boosting with minimum margin guarantees

Many classification algorithms achieve poor generalization accuracy on “noisy” data sets. We introduce a new non-convex boosting algorithm BrownBoost-δ, a noiseresistant booster, that is able to significantly increase accuracy on a set of noisy classification problems. Our algorithm consistently outperforms the original BrownBoost algorithm, AdaBoost, and LogitBoost on simulated and real data. These results hold even when early stopping of convex boosters is employed, suggesting that convex boosters may be underfitting, not overfitting. The increase in performance is correlated with the novel soft-margin maximizing parameter δ. Furthermore, we find that BrownBoost-δ is able to increase the margin at the decision boundary while sacraficing margin of easily classified examples. This tradeoff may provide a theoretical margin-based justification of empirical increased accuracy.